System and method for determining ranges to a target behind a transparent surface

ABSTRACT

Systems and methods for determining ranges to a target disposed behind a transparent surface are described. A target acquisition system receives a plurality of lidar returns, at least some of which are from a target and at least some of which are from a transparent surface. The lidar returns correspond to a portion of a lidar signal generated by a lidar, directed toward the target, and reflected back to the lidar from either the target or the transparent surface. A range measurement for each of the plurality of lidar returns is determined. The target acquisition system generates a histogram of the range measurements. The histogram includes an array including a plurality of range bins. Each range bin defines a unique portion of a predetermined distance out from the lidar. The histogram further includes a count associated with each respective range bin. The count corresponds to a number of range measurements falling within the unique portion of the predetermined distance corresponding to that respective range bin. In some implementations of the invention, the target acquisition system determines which of the range measurements correspond to the target based on the histogram. In some implementations of the invention, the target acquisition system determines which of the range measurements correspond to the transparent surface based on the histogram.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a continuation application of U.S. patentapplication Ser. No. 14/732,656, which was filed on Jun. 6, 2015, nowU.S. Pat. No. 9,829,578, which is incorporated herein by reference inits entirety.

FIELD OF THE INVENTION

The invention is generally related to detecting targets, includingfaces, in an uncontrolled environment, and more particularly, todetermining a range to a target located behind a transparent surface,such as glass.

BACKGROUND OF THE INVENTION

Conventional facial detection and recognition techniques attempt tolocate and acquire a target, such as a face, in an uncontrolledenvironment. In some of such environments, a transparent surface may bedisposed between the target and an acquisition system, such as a lidar(i.e., laser radar). For example, the target may be on an other side ofa transparent storefront in a retail environment, behind a windshield orother window in a vehicle checkpoint environment, or behind some othertransparent surface in another environment as would be appreciated.

In such environments, the acquisition system may receive return signalsfrom the transparent surface, from material on the transparent surface,from the target, from other objects, or any combination thereof.Determining which of these return signals corresponds to measurements ofa range to the target, as opposed to the transparent surface, etc., isdifficult.

What is needed is an improved system and method for determining range toa target located behind a transparent surface.

SUMMARY OF THE INVENTION

Various implementations of the invention relate to determining range toa target located behind a transparent surface. In some implementaions ofthe invention, a target acquisition system receives a plurality of lidarreturns, at least some of which are from a target and at least some ofwhich are from a transparent surface. The lidar returns correspond to aportion of a lidar signal generated by a lidar, directed toward thetarget, and reflected back to the lidar from either the target or thetransparent surface. A range measurement for each of the plurality oflidar returns is determined. In some implementations of the invention,the target acquisition system generates a histogram of the rangemeasurements. Such a histogram includes an array including a pluralityof range bins. Each range bin defines a unique portion of apredetermined distance out from the lidar. The histogram furtherincludes a count associated with each respective range bin. The countcorresponds to a number of range measurements falling within the uniqueportion of the predetermined distance corresponding to that respectiverange bin. In some implementations of the invention, the targetacquisition system determines which of the range measurements correspondto the target based on the histogram. In some implementations of theinvention, the target acquisition system determines which of the rangemeasurements correspond to the transparent surface based on thehistogram.

These implementations, their features and other aspects of the inventionare described in further detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a target acquisition system and an environment inwhich it operates according to various implementations of the invention.

FIG. 2 illustrates a histogram useful to target acquisition systemaccording to various implementations of the invention.

FIG. 3 illustrates an operation of the target acquisition systemaccording to various implementations of the invention.

DETAILED DESCRIPTION

Detecting and subsequently identifying (or recognizing) a face of atarget in an uncontrolled environment is challenging, especially in anuncontrolled outdoor environment. First, the target is free to moveinto, out of, and within a field of view of the camera, at a variety ofranges and any number of other motion factors as would be appreciated.Second, illumination of the target differs by weather, time of day,orientation of the target, objects in the environment, and any number ofother illumination factors as would be appreciated. Third, having thetarget inside a vehicle dramatically increases the challenges byintroducing vehicle type, vehicle motion, location of the target in thevehicle, window tinting, reflections, sunroofs, interior lighting, andany number of other vehicle factors as would be appreciated. Otherfactors provide further challenges to detecting faces in theuncontrolled environment.

FIG. 1 illustrates a target acquisition system 100 according to variousimplementations of the invention. In some implementations of theinvention, target acquisition system 100 includes a lidar system 120. Insome implementations of the invention, lidar 120 is a dual chirp lidarthat includes two or more beams, each of which provides measurements ofa range and a Doppler velocity for each of various points on a surfaceof a target. Such a multi-beam, dual chirp lidar is available fromDigital Signal Corporation, Chantilly, Va., and is described in U.S.Pat. No. 8,717,545 to Sebastian et al., which is incorporated herein byreference in its entirety.

In some implementations of the invention, target acquisition system 100combines lidar 120 with a video camera 130. In some implementations ofthe invention, camera 130 includes a digital camera. In someimplementations of the invention, camera 130 includes a digital videocamera. In any of the above-described implementations of the invention,camera 130 includes an infrared camera. Camera 130 captures and outputsone or more acquired images (sometimes referred to as an image stream)of a scene. Images 135 may capture a target 110, or a face of target110, in the scene as would be appreciated. Such a combined lidar 120 andvideo camera 130 is also available from Digital Signal Corporation,Chantilly, Va., and is also described in U.S. Pat. No. 8,717,545.

In some implementations of the invention, target acquisition system 100comprises a face detection system 140. In some implementations of theinvention, face detection system 140 detects a face (or other target) inthe scene, and attempts to obtain a three-dimensional image (i.e., acollection of three-dimensional measurements) of the face based on therange and Doppler velocity measurements from lidar 120. In someimplementations of the invention, face detection system 140 detects aface (or other target or other feature of a target) in the scene, andattempts to obtain a three-dimensional image of the face based on therange and Doppler velocity measurements from lidar 120 and images fromcamera 130.

In some implementations of the invention, face detection system 140 maycomprise various hardware, software, firmware and/or any combinationthereof that may be configured to perform various functions, includingthe functions described herein, as would be appreciated. Once soconfigured, facial detection system 140 becomes a particular machineconfigured to implement various features and aspects of the invention aswould be appreciated. In some implementations of the invention, facialdetection system 140 includes a computing processor and a memory (nototherwise illustrated), where the memory is configured to storeinstructions that, when executed by the computing processor, implementand/or perform various features and aspects of the invention, again, aswould be appreciated.

In some environments, target 110 is disposed on the other side of atransparent surface 160 from target acquisition system 100. In someimplementations of the invention, transparent surface 160 may includevarious types of transparent glass, plastic, or similar transparent orsemi-transparent materials as would be appreciated. In someimplementations of the invention, transparent surface 160 may betransparent to frequencies associated with lidar 120 but not necessarilyvideo camera 130 as would be appreciated. In some implementations of theinvention, transparent surface 130 may be a windshield or other windowof a vehicle. In some implementations of the invention, transparentsurface 130 may be an exterior transparent surface (e.g., window, door,etc.) of a building such as, but not limited to, an office, a house, arestaurant or other building as would be appreciated. In someimplementations of the invention, transparent surface 130 may be aninterior transparent surface in a building such as, but not limited to,an interior office window, an interior door, a partition, a screen, awall of a security, a wall of a customs area, or other interiortransparent surface as would be appreciated.

As discussed above, in the environment illustrated in FIG. 1, lidar 120may receive lidar returns from transparent surface 160, from target 110,or from both transparent surface 160 and target 110, and/or otherobjects in the environment (not otherwise illustrated). In someenvironments, lidar 120 may also receive lidar returns from dirt 170 ontransparent surface 160. As would be appreciated, lidar 120 generates arange measurement for each of the received lidar returns. Depending uponthe environment, target 110 may be a few inches or several feet fromtransparent surface. The variety of the lidar returns in connection withan unknown proximity of lidar 120 to transparent surface 160 and totarget 110, as well as an unknown proximity of target 110 to transparentsurface, may make detecting target 110, and subsequently obtaining anaccurate three-dimensional image of target 110 difficult. Moreparticularly, the variety of possible lidar returns makes it difficultto determine which lidar returns belong to transparent surface 160 andwhich belong to target 110.

According to various implementations of the invention, face detectionsystem 140 utilizes a histogram 200 such as that illustrated in FIG. 2.Histogram 200 may comprise an array 210 including a plurality of rangebins 220. An anticipated maximum range between lidar 120 and target 110is typically known or specified. This maximum range may be divided intoa number of range segments, each having a range width corresponding to aportion of the maximum range. Each range bin 220 may be assigned to oneof the range segments, and then array 210, which includes all of rangebins 220, corresponds to an extend of the entire maximum range. Whilenot illustrated in FIG. 2, array 210 may have a non-zero minimum rangein instances where target 110 is not expected at ranges closer than thenon-zero minimum range as would be appreciated.

As each range measurement is generated, face detection system 140 placesthe range measurement in an appropriate range bin 220 based on the rangesegment within which the range measurement falls. In someimplementations of the invention, in doing so, face detection system 140increments a counter associated with the appropriate range bin 220. Inthis manner, range measurements are sorted and counted based on therange bin 220 within which they fall. As illustrated in FIG. 2, a numberof range measurements fall within range bins 220A, 220B, 220C and 220Das evidenced by their corresponding counters 230A, 230B, 230C and 230D,respectively. In some implementations of the invention, a lidar returnreceived by lidar 120 may have a signal-to-noise ratio that exceeds acertain threshold before the range measurement associated with thatlidar return is added to histogram 200.

As additional range measurements are generated, sorted and counted intoarray 210, clusters of range bins 220 may begin to form. For example, inFIG. 2, a first cluster of range bins formed corresponds to bin 220A anda second cluster of bins formed corresponds to bins 220B, 220C, and220D. As illustrated, in some, though not all, implementations of theinvention, range measurements associated with transparent surface 160tend to fall within a relatively fewer number of range bins 220; whereasrange measurements associated with target 110 tend to fall within arelatively greater number of range bins 220. This is as expected becausetransparent surface 160 is typically thin and typically uniform relativeto a surface of target 110.

According to various implementations of the invention, a nearest clusterin histogram 200 is deemed to be those range measurements associatedwith transparent surface 160, and a second nearest cluster in histogram200 is deemed to be those range measurements associated with target 110.In some implementations of the invention, where transparent surface 160generates few, if any, range measurements and only a single cluster isformed in histogram 200, this single cluster may be deemed to be thoserange measurements associated with target 110.

According to various implementations of the invention, once a cluster ofbins in histogram 200 is deemed to be associated with target 110, rangemeasurements outside this cluster may be filtered as extraneous and insome implementations, ignored.

In some implementations of the invention, a cluster corresponds to thosecontiguous bins 220 in histogram 200 for which corresponding counter 230exceeds a certain threshold. As would be appreciated, lidar 120 may besubject to noise, which may result in spurious range measurements beingsorted and counted into histogram 200. In an effort to reduce anynegative impact of such noise, in some implementations of the invention,only those bins whose counter exceeds a certain threshold may beconsidered for purposes of clustering as would be appreciated.

In some implementations of the invention, other surfaces may also bedisposed between lidar 120 and target 110. For example, when target 110is inside a vehicle, lidar 120 may receive returns from other vehiclecomponents such as, but not limited to, a sun visor, a dashboard, asteering wheel, etc. These vehicle components may result in rangemeasurements that also form clusters in histogram 200. In many cases,these vehicle components will reside closer to transparent surface 160as opposed to target 110 and may be discriminated accordingly. However,in some cases, such as when these vehicle components include neck rests,seat backs, back seats, rear windows, etc., target 110 may reside closerto transparent surface 160 than such vehicle components as would beappreciated. Of course, even with clusters corresponding to such vehiclecomponents in histogram 200, target 110 should typically correspond tothe second nearest cluster in histogram 200.

Various implementations of the invention attempt to locate, detect andfocus on target 110 before a high resolution or high quality image oftarget 110 is generated. Various implementations of the inventionattempt to locate target 110 on the other side of transparent surface160 in order to subsequently determine and provide an optimal set ofcamera settings at the onset of (and in some implementations during)acquisition of a high quality image of target 110, such as athree-dimensional image of the face, as described in co-pending U.S.patent application Ser. No. 14/732,657, entitled “System and Method forIntelligent Camera Control,” filed on even date herewith, and which isincorporated herein by reference in its entirety.

In some implementations of the invention, target acquisition system 100may operate in a detection phase during which target acquisition system100 detects target 110 in the environment and an acquisition phaseduring which a high quality image of target 110 is acquired or captured.In some implementations, during the detection phase, lidar 120 may notscan its beams until target is detected and optimal camera settings forvideo camera 130 are determined. Rather than scan its beams, lidar 120may simply direct two or more beams toward a particular region in theenvironment and attempt to detect target 110. In some implementions,during the acquisition phase, lidar 120 scans target 110 while videocamera 130, adjusted with optimal camera settings, captures images oftarget 110 to obtain a high quality three dimensional image of target110. One problem associated with lidar 120 not scanning its beams duringthe detection phase is that the beams of lidar 120 may be directed todirt 170 or some other non-transparent material disposed on transparentsurface 160. In order to accommodate for such a contingency, in someimplementations of the invention, lidar 120 directs its beams at a firstspot on transparent surface 160, gathers a number of range measurements,directs its beams to a second spot on transparent surface 160, where thesecond spot is a few centimeters or a few inches from the first spot,and gathers a number of additional range measurements. This may berepeated any number of times as would be appreciated. In someimplementations of the invention, lidar 120 directs its beams to threeseparate spots on transparent surface 160 in order to avoidcomplications caused by dirt 170. As would be appreciated, the spots maybe separated vertically, horizontally, or a combination thereof.

FIG. 3 illustrates an operation 300 of face detection system 140according to various implementations of the invention. In an operation310, face detection system 140 receives a plurality of lidar returnsfrom lidar 120. In some implementations of the invention, these lidarreturns correspond to portions of lidar signals reflected back fromtransparent surface 160 or target 110 or both and received by lidar 120.

In an operation 320, a range measurement for each of the received lidarreturns is determined. In some implementations of the invention, lidar120 determines these range measurements and provides them to facedetection system 140. In some implementations of the invention, lidar120 forwards the lidar returns to face detection system 140 and facedetection system 140 determines the corresponding range measurements.

In an operation 330, face detection system 140 generates a histogram 200of the range measurements. According to various implementations of theinvention, histogram 200 includes array 210 of range bins 220, whereeach range bin 220 corresponds to a unique portion of the anticipatedmaximum distance between lidar 120 and target 110, and where each rangebin includes a counter indicative of the number of range measurementsthat fall within that range bin 220.

In an operation 340, face detection system 140 determines, based onhistogram 200, which range measurements correspond to target 110. Insome implementations of the invention, in order to determine which rangemeasurement correspond to target 110, face detection system 140identifies clusters that are formed in histogram 200, where each clustercorresponds to one or more adjacent range bins 220 each of which'scounter exceeds a predetermined threshold, as would be appreciated. Insome implementations of the invention, face detection system 140determines the nearest cluster (i.e., the cluster of range bins 220having a range closest to lidar 120) as corresponding to transparentsurface 160 and determines the second nearest cluster (i.e., the clusterof range bins 220 having a range second closest to lidar 120) ascorresponding to target 110. In some implementations of the invention inwhich transparent surface 160 provides little, if any, lidar return andhence no cluster is formed in histogram 200, face detection system 140determines the nearest cluster as corresponding to target 110 as wouldbe appreciated. In some implementations of the invention, otherdeterminations may be made depending on the environment and in whichcluster target 110 might be expected.

In some implementations of the invention, in a subsequent operation (nototherwise illustrated), face detection system 140 filters rangemeasurements corresponding to target 110 thereby in effect eliminatingrange measurements not corresponding to target 110.

While the invention has been described herein in terms of variousimplementations, it is not so limited and is limited only by the scopeof the following claims, as would be apparent to one skilled in the art.These and other implementations of the invention will become apparentupon consideration of the disclosure provided above and the accompanyingfigures. In addition, various components and features described withrespect to one implementation of the invention may be used in otherimplementations as well.

What is claimed is:
 1. A method for determining ranges to a targetdisposed behind a transparent surface, the method comprising: receivinga plurality of lidar returns, at least some of which are from a targetand at least some of which are from a transparent surface, the lidarreturns corresponding to a portion of a lidar signal generated by alidar, directed toward the target, and reflected back to the lidar fromeither the target or the transparent surface, wherein receiving aplurality of lidar returns comprises: receiving a first plurality oflidar returns from the lidar when one or more lidar beams are directedtoward a first spot; causing the one or more lidar beams of the lidar tobe directed toward a second spot; and receiving a second plurality oflidar returns from the lidar when one or more lidar beams are directedtoward the second spot; determining a range measurement for each of theplurality of lidar returns; generating a histogram of the rangemeasurements; and determining which of the range measurements correspondto the target based on the histogram.
 2. The method of claim 1, furthercomprising: determining which of the range measurements correspond tothe transparent surface based on the histogram.
 3. The method of claim1, wherein the histogram comprises a plurality of range bins and whereindetermining which of the range measurements correspond to the targetbased on the histogram comprises: arranging the plurality of range binsof the histogram into one or more clusters of adjacent range bins havingan associated count greater than a predetermined threshold.
 4. Themethod of claim 3, wherein determining which of the range measurementscorrespond to the target based on the histogram comprises: arranging theplurality of range bins of the histogram into two or more clusters ofadjacent range bins, and wherein determining which of the rangemeasurements correspond to the target based on the histogram comprisesdetermining that the range measurements associated with a second nearestone of the two or more clusters correspond to the target.
 5. The methodof claim 4, further comprising determining that the range measurementsassociated with a nearest one of the two or more clusters correspond tothe transparent surface.
 6. A system for determining ranges to a targetdisposed behind a transparent surface comprising: a lidar configured todirect a lidar signal toward a target and to receive a plurality oflidar returns, the plurality of lidar returns corresponding to a portionof the lidar signal reflected back to the lidar from either the targetor a transparent surface, wherein at least some of the plurality oflidar returns are from the target and at least some of the plurality oflidar returns are from the transparent surface, wherein the lidarconfigured to receive a plurality of lidar returns comprises the lidarconfigured to: receive a first plurality of lidar returns from the lidarwhen one or more lidar beams are directed toward a first spot; receive asecond plurality of lidar returns from the lidar when one or more lidarbeams are directed toward a second spot; and a processor configured to:determine a range measurement for each of the plurality of lidarreturns, generate a histogram of the range measurements, and determiningwhich of the range measurements correspond to the target based on thehistogram.
 7. The system of claim 6, wherein the processor is furtherconfigured to: determine which of the range measurements correspond tothe transparent surface based on the histogram.
 8. The system of claim6, wherein the histogram comprises a plurality of range bins and whereinthe processor configured to determine which of the range measurementscorrespond to the target based on the histogram comprises the processorconfigured to arrange the plurality of range bins of the histogram intoone or more clusters of adjacent range bins having an associated countgreater than a predetermined threshold.
 9. The system of claim 8,wherein the processor configured to determine which of the rangemeasurements correspond to the target based on the histogram comprisesthe processor configured to arrange the plurality of range bins of thehistogram into two or more clusters of adjacent range bins, and whereinthe processor configured to determine which of the range measurementscorrespond to the target based on the histogram comprises the processorconfigured to determine that the range measurements associated with asecond nearest one of the two or more clusters correspond to the target.10. The system of claim 9, wherein the processor is further configuredto determine that the range measurements associated with a nearest oneof the two or more clusters correspond to the transparent surface.